PRECEDING PAGE BLANK NOT FILMED 


N89 - 19823 


EMMA : THE EXPERT SYSTEM FOR 
MUNITION MAINTENANCE 


Barry E. Mullins, Capt, USAF 
Air Force Armament Laboratory (AFATL/FXG) 
Eglin Air Force Base, Florida 32542-5434 


ABSTRACT zt ms M u r i : i t n "i i ; n t er> a n- - e w i 1 1 surely 

suffer as a co u sec) u enoe of this technol- 
EMMA (Expert Missile Maintenance Aid) ogy. The technology advances that have 

is the result of research sponsored by the improved the effectiveness of munitions 

Air Force Armament Laboratory, Air Force are simultaneously complicating the main- 

Systems Command at Eglin Air Force Base tenance of these munitions by increasing 

(AFB) , Florida. It is a first attempt to the functionality of the munition typical- 

enhance maintenance of a tactical munition ly making the munition harder to maintain, 

at the field and depot level by using Munition test equipment and associated 

artificial intelligence (AI) techniques. software do not adequately diagnose 

The ultimate goal of EMMA is to help a faults. Automatic test equipment (ATE) is 

novice maintenance technician isolate and plagued by high false alarm rates. Guid- 

diagnose electronic, electromechanical, ance and control sections returned to the 

and mechanical equipment faults to the depot are currently experiencing approxi- 

board/chassis level more quickly and con- mately 28 to 44% retest OKs. Many faults, 

sistently than the best human expert using as many as one out of every four, cannot 

the best currently available automatic be detected by ATE. Sequential testing is 

test equipment (ATE). To this end, EMMA typically how ATE performs diagnostic 

augments existing ATE with an expert sys- testing. This limits the diagnostic capa- 

tem that captures the knowledge of design bilities of the test equipment. Addition- 

and maintenance experts. ally, ATE cannot diagnose beyond multiple 

linked components. 

This paper describes the EMMA pro- 
gram. It addresses such issues as how the Another aspect of munition mainte- 

field-level expert system prototypes were nance that must be addressed when dealing 

evaluated as well as the results of the with munition reliability is the personnel 

evaluations. Additionally, current work shortage. The current shortage of skilled 

on the depot-level prototypes is discussed munition maintenance technicians is a 

as well as issues related to using DOD- serious problem. Demographic projections 

STD-2167 to document the development of indicate that this dilemma will not sub- 
expert systems. This paper will briefly side in the near future. Since experi- 

address several study tasks performed enced technicians are able to diagnose a 

during EMMA. The paper concludes with a fault quicker and more reliably than a 

discussion of future plans for a follow-on novice, the knowledge acquire by the expe- 

program and other areas of concern. rienced technician (expert) throughout the 

years should be captured so that this 
INTRODUCTION knowledge can be used by novice techni- 

cians during future diagnostic sessions. 

Weapon systems of today are undoubt- 
edly benefiting from technology advances Artificial intelligence (AI) technol- 
and justifiably so. Munitions are becom- ogy is one approach to increasing the 

ing more sophisticated and "smarter" as a reliability and maintainability of ex- 
result of this technology. Electronically isting and future weapon systems. One 

sophisticated munitions are quickly infil- popular and heavily cited definition of 

trating the Department of Defense arsenal artificial intelligence is provided by Dr. 

of weapons. Simple bombs are becoming Elaine Rich, University of Texas at 

relics of the past. Austin. She defines AI as follows: "Arti- 

ficial intelligence is the study of how 
However, some shortcomings can be computers do things at which, at the mo- 

associated with incorporating new technol- ment, people are better" (Rich, 1983:1). 

ogy into current and future weapon sys- A subset of AI is a field called expert 
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systems. This area of AI has emerged 
recently with the greatest amount of suc- 
cess (Hayes-Roth et al., 1983:xi). Donald 
Waterman defines expert systems as "so- 
phisticated computer programs that manipu- 
late knowledge to solve problems effi- 
ciently and effectively in a narrow prob- 
lem area" (Waterman, 1986:xvii). 

The Air Force has recognized the 
importance of increasing munition reli- 
ability. In a joint memorandum. General 
Gabriel, former Air Force Chief of Staff, 
and Verne Orr, former Secretary of the Air 
Force stated, "For too long, the reli- 
ability and maintainability of our weapon 
systems have been secondary considerations 
in the acquisition process. It is time to 
change this practice ...." 

EMMA 


EMMA (Expert Missile Maintenance Aid) 
is the result of research sponsored by the 
Air Force Armament Laboratory, Air Force 
Systems Command at Eglin AFB, Florida and 
is a first attempt to enhance Air Force 
tactical munition maintenance by applying 
artificial intelligence/expert system 
technology to ATE. The objective of EMMA 
is to develop an automated smart munition 
test system that augments existing ATE. 
The ultimate goal of EMMA is to help a 
novice munition technician isolate and 
diagnose electronic, electromechanical, 
and mechanical equipment faults to the 
board/chassis level more quickly and con- 
sistently than the best human expert using 
the best currently available ATE. 

EMMA is a thirty month effort split 
into two phases. Phase 1 began in Septem- 
ber 1986 and concluded 10 months later in 
July 1987. This phase addressed the 
field-level maintenance of tactical muni- 
tions and ultimately resulted in two 
field-level expert system prototypes. 
Phase 2 began in August 1987 and is sched- 
uled to conclude in April 1989, 20 months 
later. Phase 2 focuses on depot-level 
maintenance and will produce two depot- 
level expert system prototypes. Since 
depot-level diagnostic activities are more 
in-depth and detailed than the field, this 
phase is expected to be more difficult and 
of greater complexity. This accounts for 
the greater time allotted to this phase. 
The prototypes from both phases are tar- 
geted towards the maintenance technicians. 
Since EMMA is constrained by schedule and 
money, the number of tests developed under 
this effort is limited, yet sufficient to 
demonstrate concept feasibility of using 
expert systems for munition maintenance. 

EMMA draws on many different types of 
knowledge and information to perform the 
diagnosis of the faulty munition. The 
EMMA knowledge base consists of mainte- 
nance rules or Technical Orders (TOs), 


maintenance technician practices (heuris- 
tics), Unit Under Test (UUT) design, ex- 
isting test equipment capabilities, fail- 
ure rates, and test costs. This knowledge 
is gleaned from various sources including 
TOs, schematics of the UUT and the test 
equipment, knowledge acquisition inter- 
views with munition maintenance experts 
and the experts that designed the munition 
and test equipment. Figure 1 depicts how 
this knowledge is brought to bear on the 
problem of diagnosing the faulty munition. 
First, the symptoms are derived from the 
test equipment and technician observa- 
tions. This information is supplied to 
the expert system via a sophisticated, 
user-friendly interface. The expert sys- 
tem then employs the knowledge stored in 
the knowledge bases and derives a repair 
strategy which is displayed to the techni- 
cian using the EMMA computer. 



EMMA is a dual contract effort per- 
formed by Raytheon Company, Missile Sys- 
tems Division in Bedford Massachusetts, 
and Rockwell International Corporation, 
Autonetics Sensors and Aircraft Systems 
Division in Anaheim, California. Both 
contractors will develop a phase 1 and 
phase 2 EMMA prototype resulting in a 
total of four prototypes. Both contrac- 
tors were allowed to select their candi- 
date vehicle for the EMMA program within 
specified limits. Raytheon selected the 
AIM-7F Sparrow missile as their candidate 
munition. Rockwell chose the GBU-15 modu- 
lar glide bomb. 

THE AIM-7F FIELD-LEVEL EMMA PROTOTYPE 

The Raytheon field-level (phase 1) 
EMMA prototype was designed to enhance the 
field-level maintenance of the AIM-7F 
missile by augmenting the missile's test 
set. All references to the word "EMMA" in 
this section refer to the Raytheon AIM-7F 
version of EMMA. The field-level test set 
for the AIM-7F is the AN/DSM-162 test set. 
EMMA is hosted on a Symbolics 3670 LISP 
machine running the expert system shell 
ART (Automated Reasoning Tool). ART pro- 
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vides a production language that is pri- 
marily rule-based. Consequently, EMMA was 
developed using the rule-based approach. 
The Symbolics computer is connected to the 
AN/DSM -162 test set via an RS-232 cable. 
Figure 2 illustrates the major components 
of the EMMA system and how they are inter- 
connected . 



PRINTER 


Figure 2. AIM-7F Field-level 
EMMA Prototype 

The RS-232 cable allows EMMA to oper- 
ate in three modes -- automatic, semi- 
automatic, and manual. The distinguishing 
characteristics of these modes is the 
level of automation EMMA is allowed during 
the diagnostic session. The automatic 
mode uses the RS-232 interface to allow 
EMMA to direct the diagnostic testing and 
resequencing of tests. EMMA automatically 
accepts data from the test set via the RS- 
232 cable, performs the fault isolation, 
and directs the test set to perform addi- 
tional test, if required, until the fault 
is detected or all tests pass. If a fault 
is detected during automatic operation, 
the user may switch to semi-automatic mode 
for closer control over the testing and 
the ability to query after each test seg- 
ment. 

The semi-automatic mode operates sim- 
ilar to the automatic mode with one excep- 
tion. This mode stops execution of EMMA 
at the completion of each unique test 
segment. This allows the technician to 
query EMMA recommendations using the ex- 
planation capability. Another advantage 
of the automatic modes (semi and full) is 


data integrity. Since EMMA passes the 
data between the test set and the Symbol- 
ics computer via the RS-232 cable, the 
data are more likely to remain valid as 
opposed to transferring data via a techni- 
cian who could inadvertently introduce 
errors . 

The last mode is manual. This mode 
is provided in case an RS-232 connection 
is not possible. As the name implies, all 
activities between the test set and the 
Symbolics computer must be performed man- 
ually by the technician. EMMA will direct 
the technician to perform the appropriate 
actions to the test set and wait for the 
response. The technician enters the re- 
sponses from the test set into EMMA. 

As with most expert systems, EMMA is 
able to explain its reasoning process to 
the user (technician). EMMA explains its 
fault detection and resequencing logic. 
In other words, EMMA explains a detected 
fault and why a certain test is being 
recommended. Two levels of explanation 
are available depending upon the experi- 
ence of the technician. The technician 
may request an explanation during any 
phase of the diagnostic process. This 
allows the technician to query EMMA during 
a consultation which heightens the techni- 
cian's understanding of what EMMA is doing 
while simultaneously providing the techni- 
cian with a valuable training aid. 

One of the most critical aspects of 
any software system is its user-friendli- 
ness. If the system is difficult to use 
and the user does not use it, it has 
failed. EMMA uses windows to relay infor- 
mation to the technician and accepts in- 
formation via menus. Using a mouse, the 
technician is able to enter data quickly 
and accurately without having to learn 
cryptic commands. The majority of the 
data entered into EMMA by the technician 
is done using the mouse; however, some 
keyboard input is required. Figure 3 
shows the screen of a Symbolics computer 
running EMMA. 

THE GBU-15 FIELD-LEVEL EMMA PROTOTYPE 

The Rockwell field-level (phase 1) 
EMMA prototype was designed to enhance the 
field-level maintenance of the GBU-15 
glide bomb by augmenting the field-level 
test set — GJM-55. All references to the 
word "EMMA" in this section refer to the 
Rockwell GBU-15 version of EMMA unless 
stated otherwise. EMMA is hosted on a IBM 
PC/AT compatible computer running the 
expert system shell M.l. Although, the 
M.l language is primarily rule-based, EMMA 
was developed using an object oriented 
approach. The rules of the knowledge base 
reference objects and object attributes. 
This EMMA did not support the capability 
for an automatic mode due to hardware 
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TEST04 PRELAUNCH TUNING, PD. MISSILE READY 


TEST DESCRIPTIONS 


TEST B4 

(THE REFERENCED RF IS PD.) 

RFTER THE HERTER CYCLE HRS BEEN COMPLETED, THE 
RISSZLE IS RERDY TO TUNE. THE RPPLIEO THREE PHRSE 
4 MM2 IS CONVERTED TO «2S.0 VOC (KEEP ALIVE VOLTAGE) 
FOR HISSILE POUER RI1D INTO *60 VOC FOR GYRO SNAP 
START POUER. THE KEEP ALIVE ♦25. B VDC PROVIDES 
CONDITIONING TO SELECTIVE HISSILE CIRCUITRY DURING 
THE TUNE, DOftnRHT HMD LAUNCH TO SEPARATION HISSILE 
NODES. 

THE 2 SECOND TUNE TIHER IS ACTIVATED BY THE KEEP 
ALIVE VOLTAGE (KR), THE KA VOLTAGE RETURHED FROM THE 

UP DOUN 


Help Why [Start/Co7itint^c| EMERGENCY LcvehEXPERT User's Manual Quit 


MANUAL PROCEDURES 


TEST RESULTS 


Send configuration thumbwheels 0 3 10 
to test set 

Send MISSILE TEST RESET message 
WAITING 20 SECONDS. 

(MENU Disabled) 


RUR serial number: RBT1234 

Flight Control serial number: R-41B16 

Target Seeker serial number: R-41016 

CONFIG. B11B 

01 8651 3 

01 BB122 

82 BBB07 
03 BBB8G 

*04 0008 7 NLT 922 
*05 B0887 NLT 922 
01 BB513 
81 BB122 
62 BBB07 

83 80006 

*04 09087 NLT 922 
*05 0B007 NLT 922 


Figure 3. AIM-7F EMMA Screen 


limitations thereby leaving only the man- 
ual mode (i.e., no connecting cable). 
Figure 4 illustrates the major components 
of the EMMA system and how they are inter- 
connected . 

Uncertainty is addressed in this 
version of EMMA. When EMMA asks the tech- 
nician for information, the technician may 
enter "unknown" as a response. EMMA will 
accommodate this response by adapting its 
reasoning process using uncertainty. Un- 
certainty is handled using a MYCIN-like 
representation. When a recommendation is 
displayed to the technician, the certainty 
of the recommendation is also displayed to 
indicate the belief of the recommendation. 

The GBU-15 EMMA also possesses expla- 
nation capabilities. The technician may 
ask EMMA for an explanation or help at any 
time. EMMA will respond with either an 
explanation of the reasoning process or 
information that will guide the technician 
through the consultation. The explanation 
capability can handle queries regarding 
the reason a certain conclusion was 
reached or why EMMA is asking for informa- 
tion. As with the AIM-7F EMMA, the GBU-15 
EMMA has two levels of explanation to 
accommodate the needs of different techni- 
cians. The same training benefits exist 
in the GBU-15 EMMA as the AIM-7F EMMA. 



CURRENT 

SITUATION 



EMMA exploits the use of pull-down 
menus and function keys on the computer to 
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Figure 4. GBU-15 Freld-level 
EMMA t-rototype 





Verificatioh of EMMA 


make it as user friendly as possible. The 
majority of technician interaction with 
EMMA is performed using the keyboard. The A uniqufe^aspect of the verification 

technician typically responds to EMMA of the EMMA program is that it uses DOD- 

questions and requests with short answers STD-2167, the Defense System Software 

thereby reducing the probability of erro- Development standard, to develop the ex- 

neous data being entered. Figure 5 shows pert system prototypes. This is one of 

the screen of the computer running EMMA. the first attempts to apply this standard 

to the development of an expert system. 
E VALUATION OF THE EMMA PROTOTYPES DOD-STD-2167 traditionally employs top- 

down development of large software sys- 
Meaningful evaluation of expert sys- terns. Expert systems on the other hand 

terns has been an often discussed but sel- are developed using a development method- 

dom achieved topic within recent years. ology that is less rigidly defined which 

More often than not quantitative metrics typically entails an iterative approach to 

are simply not available or meaningful as software development. Additionally, ex- 

an evaluation measure. Since an expert pert systems are usually created by a 

system encapsulates the knowledge of a relatively small team, 

given expert in a given field, the effec- 
tive evaluation of the expert system may The EMMA program has shown that ex- 

be difficult at best. Validation must be pert systems can be developed using DOD- 

used to justify the representation levels STD-2167 software development require- 

of expert systems (O'Keefe et al., 1987). ments. With some careful tailoring of 

some of the documents, this standard can 
Validation is typically considered a be effectively used to document the pro- 
part of evaluation, and evaluation is gram and provide the program manager valu- 

concerned with determining the comprehen- able insight into the contractor's soft- 

sive value of an expert system (O'Keefe et ware development, testing, and evaluation 

al., 1987). Validation should not be efforts. The tailored documents were 

confused with verification. "Validation altered to accommodate the iterative na- 

refers to building the right system (that ture of expert system development, 

is, substantiating that a system performs 

with an acceptable level of accuracy). Since verification must determine 

whereas verification refers to building whether an expert system correctly imple- 

the system 'right' (that is, substanti- ments its specifications, testing must 

ating that a system correctly implements occur in order to validate this require- 

its specifications)" (O'Keefe et al., ment. Again, DOD-STD-2167 proved to be 

1987). Validation and verification will adequate for verification testing once 

be addressed in this paper as they apply extended. Using the testing procedures 

to EMMA. called out in this standard, the correct 


EMMA CONFIG NO: 33 TEST NO: 0096 NOGO: INV/M6 Al 

COMPONENT CABLE CERTAINTY 

INV/CONV 60% 

CONTROL UNIT (HARNESS) UUT INPUT CABLE 053A20(BLU) 40% 

RECOMMENDATION: 

CHECK THE INPUT CABLE OF THE UNIT UNDER TEST. IF OK, THEN 
R&R INV/CONV 

JUSTIFICATION: 

A RELATED TEST WITH A DIFFERENT INPUT CHANNEL PASSED AND 
THE PARAMETERS THAT FAILED WERE APPROXIMATELY ZERO; THUS 
THE INPUT CABLES OF THE UNIT UNDER TEST COULD BE THE 
CAUSE. A MORE LIKELY FAILURE IS THE INV/CONV BECAUSE THE 
+ 28 VDC ELECTRONICS IS COMPLEX AND COULD CAUSE ZERO 
FAILURES AS INDICATED 


Figure 5. GBU-15 EMMA Screen 
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implementations of the specifications for 
EMMA were verified. Two levels of testing 
occurred to accomplish this task. First, 
informal testing took place. This testing 
verified the integrity of the individual 
computer software units before the units 
were integrated into the system and tested 
as a system. Informal testing was per- 
formed by the knowledge engineer. Since 
expert system development is iterative in 
nature, informal testing essentially oc- 
curs throughout development. The knowl- 
edge engineer and the expert verify the 
expert system and identify potential cor- 
rections and enhancements. Based on these 
recommendations, the knowledge engineer 
was able to incorporate the recommenda- 
tions. Second, formal testing occurred. 
An independent team performed the formal 
testing by exercising EMMA using test 
plans and test descriptions generated 
using DOD-STD-2167. 


Validation of EMMA 


The validation of EMMA will be ad- 
dressed in this paper in two areas. 
First, the performance validation of EMMA 
will be discussed (i.e., how well EMMA 
performed). Second, the human factors 
aspect of validation will be addressed. 
Both areas are extremely important to the 
success of an expert system. The follow- 
ing paragraphs will present the validation 
of the two EMMA prototypes. The valida- 
tion methodology will be discussed fol- 
lowed by the results of the validation. 

As with most expert systems, the 
ultimate measure of success is determined 
when the system is used by the end users 
(in this case the field-level munition 
technician). This is the approach taken 
with the EMMA program. Both contractors 
took their respective prototypes to Air 
Force bases in which their selected muni- 
tion is used and a field-level maintenance 
capability exists. This allowed the pro- 
totypes to be evaluated in an actual field 
test environment. 


After considering several alterna- 
tives, both contractors decided to use a 
toggle switch box to insert faults into a 
known good missile. This approach was 
necessary since it was feared that the 
maintenance squadrons in the field might 
not have a sufficient number of faulty 
munitions during the evaluation period. 
These faults were induced by the user by 
simply toggling one of the switches which 
in turn would disturb one or more signals 
within the munition. The faults were 
defined by the domain expert such that the 
faults would adequately exercise the var- 
ious characteristics of the EMMA proto- 
types which included the resequencing 
logic, the explanation capability, and the 
fault isolation logic. To verify the 
expert was not trying to select only the 


faults EMMA could handle best, the mainte- 
nance experts agreed the selected faults 
were representative of faults they common- 
ly experience. 

The AIM-7F EMMA Evaluation. Raytheon 
took their field level AIM-7F prototype to 
the 325th Equipment Maintenance Squadron 
(EMS) at Tyndall AFB, Florida for an eval- 
uation period that began on 8 June 1987 
and concluded on 12 June 1987. 

The argument could be made that EMMA 
should accurately diagnose all the induced 
faults since the expert system and the 
faults were derived from the same source 
— the domain expert. In order to demon- 
strate the robustness of EMMA, an addi- 
tional evaluation methodology was used. 
Two faulted missiles were saved by the EMS 
prior to the evaluation. These missiles 
had previously failed testing using the 
AN/DSM-162 test set. However, the fault 
data for these missiles were not released 
by the EMS personnel until after the EMMA 
evaluation. A third missile became avail- 
able during the EMMA evaluation by failing 
a flight line test during prelaunch tun- 
ing. This missile was an excellent exer- 
cise for the EMMA prototype since it was 
not previously tested by the AN/DSM-162 
test set. Its fault was unknown to every- 
one present at the evaluation. All three 
missiles (sometimes referred to as "mys- 
tery missiles" due to their unknown past) 
contained faults unknown to EMMA or the 
domain expert thereby exercising EMMA in 
an unpredictable manner. 

Four munition maintenance technicians 
from the EMS at Tyndall were used for the 
evaluation. Two technicians were classi- 
fied as novice with little experience with 
the AN/DSM-162 test set and its associated 
operating procedures. The other two tech- 
nicians were classified as experts with a 
substantial background in using the 
AN/DSM-162 test set. Two teams of two 
technicians were created consisting of one 
expert and one novice. One team (hereaf- 
ter referred to as the EMMA team) received 
extensive training on the operation of 
EMMA. The other team (hereafter referred 
to as the non-EMMA team) was not trained 
on the EMMA system and served as a base- 
line for the evaluation. 

Twelve faults were inserted into a 
known good missile using the toggle switch 
box. The faults were diagnosed by the 
EMMA team using the EMMA system and the 
non-EMMA team using just the AN/DSM-162 
test set and the applicable TO. Perfor- 
mance of the two teams was based on the 
level of expertise of the operator, dura- 
tion of test, and the ability to diagnose 
the fault accurately. 
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The results of this evaluation exer- 
cise were very promising. There were 
three significant results derived from the 
evaluation. First, the EMMA system oper- 
ated by the EMMA team was able to consis- 
tently diagnose the fault quicker than the 
non-EMMA team using just the AN/DSM-162 
and the TO regard l ess of the experience 
level of the EMMA operator. A time sav- 
ings of 20% was seen with the novice using 
EMMA over the expert using the AN/DSM-162. 
Second, novice technicians using the EMMA 
system significantly outperformed (better 
fault diagnoses) novice technicians using 
just the AN/DSM-162 and performed 33% 
better than expert technicians using just 
the AN/DSM-162. Finally, EMMA's explana- 
tion capabilities significantly enhanced 
the abilities of the EMMA team to deter- 
mine the reason behind each fault. 

Once EMMA's abilities were exercised 
using the induced faults, EMMA was pitted 
against the mystery missiles again with 
excellent results. The EMMA team using 
EMMA correctly isolated the faults in all 
three missiles. Only after EMMA diagnosed 
the faults was the previous testing data 
on the missiles released. EMMA's diagno- 
sis was consistent with this data. 

User acceptance of EMMA was outstand- 
ing. In fact, the technicians accepted 
EMMA's diagnosis of the missile from the 
flight line and said they would have, if 
allowed, sent the missile to the depot 
with no further testing using the AN/DSM- 
162 test set. This exemplifies EMMA's 
acceptance by the EMS maintenance person- 
nel at Tyndall AFB. The technicians found 
the system to be very user-friendly. The 
mouse and the use of menus made the system 
easier to use than the bulky and cumber- 
some TOs. Also, the explanation capabili- 
ty proved to be an effective training 
mechan i sm . 

The GBU-15 EHMA Evaluation. Rockwell 
evaluated their GBU-15 EMMA at the 4th 
Equipment Maintenance Squadron located at 
Seymour Johnson AFB, North Carolina during 
the period of 22 June through 29 June 
1987. 

Four maintenance technicians were 
used in the evaluation of the EMMA proto- 
type. Two technicians were considered 
experts with several years of experience 
with the GBU-15 test environment. The 
remaining two technicians were considered 
novices with less than 6 months of expe- 
rience. Another important distinction 
between the expert and novice technicians 
is the fact that the expert technicians 
owned personal computers and therefore 
were familiar with how computers operate 
whereas the novice technicians did not own 
computers and had never used a computer 
before the EMMA evaluation. All four 
technicians were trained on how to use the 


EMMA system. After this brief training, 
the technicians felt very comfortable 
using the system. 

Twenty-two simulated faults were in- 
duced into the known good munition with 
the intent of evaluating EMMA's capabili- 
ties to handle the following five areas: 
resolution of ambiguities between major 
shop replaceable units (SRU), referencing 
lower configuration testing to facilitate 
further component resolution, distin- 
guishing between a cable failure and a 
circuit card assembly (CCA) gain failure, 
resolution of ambiguities between CCA's, 
and recognizing operator errors or test 
set problems. Six of the twenty-two in- 
duced faults were in the all-up-round 
(AUR) configuration (i.e., the test was 
performed while the GBU-15 munition was 
completely intact). The remaining sixteen 
faults were in the control module stand 
alone configuration. EMMA was able to 
handle these five areas by analyzing addi- 
tional test parameters as well as insti- 
tuting and analyzing tests related to the 
failed test. 

The diagnostic results of the induced 
faults showed substantial time savings in 
fault isolation and increased diagnostic 
capabilities. While the munition was in 
the control module stand alone configura- 
tion, a time savings of 40% was seen over 
conventional testing with the GJM-55. 
When the munition was in the AUR configu- 
ration, EMMAwas able to provide up to 74% 
time savings. This is due to EMMA’s capa- 
bility to resolve failures while the muni- 
tion is in the AUR configuration thereby 
saving the technician from having to per- 
forming testing in stand alone configura- 
tion . 

The GJM-55 test set, in some situa- 
tions, will recommend more than one sus- 
pected failure. This group of failures is 
called an ambiguity group since the test 
set cannot resolve any further than this 
group. This is another of EMMA's capabil- 
ities that demonstrated promising perfor- 
mance as seen by the results that follow. 
EMMA also considered the possibility of a 
cable harness failure or the test set is 
failing. Based on these capabilities EMMA 
was able to significantly improve fault 
isolation. The results of the twenty-two 
simulated faults demonstrate this improve- 
ment. EMMA added a wiring harness check 
to 50% of all tests. EMMA deleted a CCA 
from an ambiguity group 40% of the time 
thereby reducing the number of CCA to be 
considered during testing. EMMA added a 
CCA to an ambiguity group 30% of the time 
to insure all potential CCA's are con- 
sidered during the testing. This indi- 
cates that the test set sometimes did not 
consider all potential CCA's. Finally, 
EMMA exchanged one suspect CCA in an ambi- 
guity group for another CCA 10% of the 
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time. The ability to manipulate the ambi- 
guity group to benefit fault isolation was 
demonstrated by EMMA and proved to be an 
effective fault isolation technique. 
These results directly support the time 
savings previously mentioned. 

The GBU-15 EMMA prototype also re- 
ceived accolades for its user friend- 
liness. The technicians used EMMA with 
comfort and found several items to be 
particularly laudable. Among these items 
was the understandabi 1 ity of EMMA. The 
explanation capability provided easy to 
understand responses. Another aspect they 
found beneficial was the addition of the 
internal wiring harness check as one of 
the reasons for a fault since this check 
is relatively "inexpensive" to perform and 
can prevent unnecessary and potentially 
costly future testing. The training po- 
tential of EMMA was also mentioned as one 
of its major assets with the shortage of 
skilled technicians in the munition main- 
tenance field. 


EMMA PHASE 2 


Both contractors are currently in 
phase 2 of the EMMA program. As previous- 
ly mentioned, phase 2 focuses on the main- 
tenance of tactical munitions at the depot 
level. More specifically, Raytheon is 
focusing on the depot-level maintenance of 
the AIM-7F. Rockwell is using the GBU-15 
as its depot-level maintenance munition. 
Phase 2 is a natural extension of phase 1 
since field-level faults are sent to the 
depot for repair. The prototypes devel- 
oped during the phase 2 effort will be 
more detailed extensions of the phase 1 
prototypes with one exception; the phase 2 
prototypes will augment the depot-level 
test sets. The depot test set for the 
AIM-7F is the AN/DPM-22 test set. The 
GBU-15 depot-level test set is CATS (Cal- 
culator Automatic Test Station). 


The depot-level prototypes will be 
implemented on the same computer hardware 
using the same expert system shells as the 
field-level prototypes. However, one dif- 
ference between the field and depot proto- 
types for both contractors is the inter- 
face between the test set and the EMMA 
computer. The Raytheon interface will 
only support one-way communication from 
the test set to the EMMA computer due to 
test set limitations. This is different 
than the two-way communication of the 
field prototype. Rockwell is using a two- 
way communication interface between the 
test set and the EMMA computer whereas the 
field prototype interface was manual. 
Both prototypes will again incorporate an 
explanation capability for the techni- 
cians . 


The evaluation of the depot proto- 
types will follow the same methodology 
used in phase 1. Each prototype will be 
evaluated at the actual depot location by 
actual depot technicians. Once again, 
both evaluations are scheduled to last 
five days and are scheduled to occur in 
February 1989. The AIM-7F prototype will 
be evaluated at the Naval Aviation Depot 
(NAVA VNDEP) in Alameda, California. The 
GBU-15 prototype will be evaluated at 
Rockwell's Missile Systems Division in 
Atlanta, Georgia since an organic depot 
capability currently does not exist. 

FOLLOW-ON PROGRAMS TO EMMA — EMMA 2 

A follow-on program will be initiated 
in early 1990 — EMMA 2. The primary 
thrust of EMMA 2 is to develop an expert 
system that is capable of diagnosing a 
family of tactical munitions at the depot 
level. The current EMMA is limited to one 
munition per prototype. EMMA 2 would 
attempt to expand the current prototype 
capabilities to include multiple munitions 
from the same family (e.g., AIM family, 
GBU family, surface-to-air family, etc.). 
EMMA 2 would draw on the best features of 
all prototypes developed in the two phases 
of EMMA to derive a robust system. 

OTHER ISSOES/OBSERVATIONS 

The following paragraphs present oth- 
er areas of interest to the EMMA program. 

Studies 

As part of the phase 1 effort, five 
studies were conducted to address areas of 
concern that could be incorporated into 
the depot-level prototypes and potentially 
in future maintenance expert systems. 
These studies included the reuse of muni- 
tion test programs, the use of the Ada 
language for expert system development and 
ATE test programs, the applicability of 
the Modular Automatic Test Equipment 
(MATE) standard to EMMA, the applicability 
of the Warner Robins Reliability Asset 
Monitor (RAM) database to EMMA and future 
maintenance expert systems, and the secu- 
rity issues of expert systems. A complete 
discussion of the results of these studies 
is beyond the scope of this paper. There- 
fore, the interested reader is referred to 
the two final reports of phase 1 (Elerin 
et al., 1987; Davis, 1987). These final 
reports are split into two volumes; the 
second volume contains a complete discus- 
sion of the results of these studies. 

Current Maintenance Philosophy 

The current munition maintenance phi- 
losophy of the Tactical Air Command (TAC) 
for field maintenance is that of fault 
detection (go/nogo testing). If a fault 
does occur in the field, the suspected 


38 



ORIGINAL PAGE IS 
OF POOR QUALITY 


faulty section of the missile is sent to 
the depot for repair. One of the driving 
factors of this philosophy is the shortage 
of skilled maintenance technicians in 
field-level maintenance. 

Since training these technicians is 
costly, TAC decided to eliminate an Air 
Force Specialty Code (AFSC) for munition 
maintenance. The deleted AFSC, 316X1L, 
was an electronics munition maintenance 
specialist. With this specialist no lon- 
ger available, munition, not electronic 
munition, specialist are diagnosing to- 
day's munitions. This tends to create 
problems. The munition specialists are 
typically not adequately trained to diag- 
nose the electronically-sophisticated mu- 
nitions of today. 

EMMA is capable of providing the 
necessary training of munition maintenance 
technicians. Using the explanation capa- 
bilities of EMMA, a technician can quickly 
become skilled at diagnosing the munition. 
Since EMMA'S knowledge is gleaned from 
diagnostic and design experts, the novice 
munition technician using EMMA will effec- 
tively be performing as if he has an ex- 
pert maintenance technician, the designer 
of the test set, the munition designer, 
and an instructor looking over his shoul- 
der during the diagnosis. Another aspect 
of EMMA that should reduce overall mainte- 
nance costs is its ability to diagnose to 
a greater component level than existing 
test sets used by today's technicians. 
This should significantly reduce the costs 
associated with shipping faulty munitions 
to the depot, since more faults can be 
isolated at the field. 

The munition depots are currently 
responsible for the majority of munition 
repairs. The field has very limited re- 
pair capabilities. This results in in- 
creased costs for overall maintenance of a 
munition system. The obvious cost is 
transportation of the munition between the 
field and the depot. However, the less 
tangible and potentially more significant 
cost is that of having the munition out of 
the inventory. This effectively reduces 
the number of missiles available for exer- 
cises or conflict. 

Future Maintenance Systems 

EMMA prototypes have proved the fea- 
sibility of applying AI to munition main- 
tenance. In future weapon systems, main- 
tenance expert systems should evolve with 
the weapon system instead of after the 
fact. This would allow the expert system 
to capture knowledge about the weapon as 
it is developed. Furthermore, the expert 
system should be incorporated directly 
into the ATE instead of augmenting the ATE 
with a separate computer system. This 
should make weapon systems of the future 


more supportable by considering the main- 
tenance aspect early in the weapon life 

cycle. 

SUMMARY 

Tactical munition maintenance of to- 
day has problems. EMMA is an attempt to 
relieve some of these problems by applying 
artificial intelligence/expert system 
technology. The results of EMMA indicate 
that this approach to munition maintenance 
has significant potential for future tac- 
tical maintenance systems. 

Corporate knowledge retention is one 
of the premium benefits of EMMA. Since 
EMMA is updated easily and it never "for- 
gets" knowledge, EMMA is an excellent tool 
for storing corporate knowledge as techni- 
cians come and go. Also, EMMA provides 
consistent, high quality diagnosis since 
it never has a "bad" day as contrasted 
with technicians. Rapid fault isolation 
and efficient manpower utilization are two 
more benefits of using EMMA. These bene- 
fits provided by EMMA will result in sub- 
stantial mission payoffs. Weapon system 
downtime will be decreased as well as 
personnel requirements and training time. 
However, the most significant payoff is 
the increase in the reliability of muni- 
tion maintenance procedures. 
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